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ABSTRACT 

An educational developnent teas, working at the 
Center for the Study of Evaluation to produce the Blenentary School 
Foraative Evaluation Kit, saw that its Mission was not si ■ ply to show 
that a product neets pre-detersined per for nance specifications but 
was also to ensure that a product can be readily adapted by the user 
to the "noisy" conditions under which it ultiaately will be used* 
People are likely to accept an innovation if it is consistent with 
their professional noras, levels of expertise, and day-to-day 
responsibilities* It is, therefore, the developer's function to 
devise a technology for solving a given problen that takes into* 
account the "intentions and behaviors of the target audience*" 
(Author/WH) 
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In the area of educational technology, the path which leads from 
Research through Development 1 to the school setting has been an elusive 
one. That It should be possible to transform available relevant know- 
ledge Into a form that encourages Improved Instructional practice Is ac- 
cepted with guarded optimism by educators; yet, as educational develop- 
ers know all too well, the means for doing so are far from perfected* 
Over $500,000,000 has been spent during the last two decades researching 
and developing materials and procedures aimed at Improving public educa- 
tion. (Gideonse, 1969) But, as Hood (1973), Eldell and Kltchel (1968), 
and many others have pointed out, the actual impact of such efforts on 
school practices has been less-than-spectacular. It is little wonder 
that the educational research and development (R and 0) community has 
turned serious attention to the problems of adoption and utilization of 
the produces they develop. 

Recent concern over the Impact of products and procedures emanating 

from educational research and development efforts has been accompanied 

2 

by a surge of research on the diffusion of educational Innovations. 
Much of this research is directed towards the formulation of frameworks 
and paradigms for the diffusion process and proposed strategies that can 
be used by R and D agencies to facilitate the adoption and utilization 



A concise definition offered by the National Science Foundation 
(1965) describes development as "...the systematic use of scientific 
knowledge directed toward the production of useful materials, devices, 
systems, or methods, including design and development of prototypes 
and processes." 

2 

Katz and others (1963) have defined diffusion as "...the accept- 
ance over time of some specific item — an idea or practice — by In- 
dividuals or groups or other adopting units, linked by specific channels 
of communication, to a social structure, and to a given system of values 
or culture." 
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of their products. For the most part, the research studies begin with 
the assumption that a particular R and D output Is both reliable and 
valid; the studies then proceed to Investigate the effectiveness of a 
given diffusion strategy to adopt and use these products under varied 
conditions. While such research has provided Insights o<r undoubted use 
to R and D agencies, the diffusion literature makes little or no mention 
of the processes by which the products are developed. 

The development of an educational product Is generally regarded as 
a narrowly prescribed set of activities aimed exclusively at devising 
the technology to solve a defined educational problem. If we follow 
this reasoning to Its logical conclusion, development then becomes sim- 
ply a process of converting research findings Into organized materials 
and procedures to fill the need defined by the findings. By this same 
reasoning, If the development team Is able to show that the resulting 
product brings about specified and Intended outcomes In one or more 
practice settings, the developers have successfully accomplished their 
mission. In other words, the objective of the development process pur- 
ports to ensure that a product "works," and thus solves the given educa- 
tional problem. The responsibility of designing support systems for the 
Innovation and for Informing practitioners of the availability of the 
new product Is by and large a domain appropriately handled by diffusion 
specialists or marketing managers. 

2 
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A question of means 

Staff members of the Center for the Study of Evaluation (CSE) Elementary 

3 

School Formative Evaluation Kit took exception to this problem-oriented view 
of the development process. We assumed an expanded Interpretation of the de- 
velopment mission suggesting that It Is not enough simply to show that a pro- 
duct meets pre-determlned performance specifications. Rather, we felt that 
a development team has the additional responsibility of ensuring that a pro- 
duct can be readily adopted by the user to the "noisy" conditions under which 
It ultimately will be used. Our reasoning stemmed from one commonsenslcal 
construct: Although a product may solve a specific educational problem, and 
the technology may work, still, target users may be reluctant to adopt and 
use the product If they are dissatisfied with the means by which the problem 
will be solved. 

It Is worth remembering at this point that social scientists, particu- 
larly those who study the way In which organizations change, have been pro- 
posing similar constructs for a number of years. The rationale is quite 
simple: People are more likely to accept an innovation if it is consistent 
with their professional norms, levels of expertise, and day-to-day responsi- 
bilities. The problem labelled "top priority" by researchers and policy 
makers may be just one among many problems encountered by practitioners 
during a normal working day. The identification of high-priority problems 
or needs is, of course, outside the educational developer's Jurisdiction. 
But it is the developer's function to devise a technology for solving a 
given problem that takes into account the "intentions and behaviors of the 
target audience." (Bennis, 1969) 



Churchman, D., Petrosko, J., 6 Spooner-SmI th, L. Elementary School 
Formative Evaluation Kit, Field Test Version, May 197*». 



The designers of the formative Evaluation Kit were thus faced with 
two objectives. First, they were to devise a technology that would enable 
an elementary-school staff to carry out a formative evaluation of a local 
educational program. The' theoretical bases for the evaluation which the 
technology was to incorporate have been described In the first paper pre- 
sented at this symposium. It was clear to all of us that the staff was 
not to develop an Instructional product per se, I.e., materials to train 
school personnel In discrete and transferable evaluation skills. What 
was required was that the Formative Evaluation Kit provide guidelines for 
day-to-day conduct of an evaluation while the target school program was 
in operation. Further, the Kl t was intended to help practicing teachers 
and administrators evaluate the implementation and progress of an exist- 
ing program without additional outside help; this last condition caused 
the Kit, to be viewed as a "consultant in a book." Additionally, the devel- 
opment task included the design of reusable materials flexible enough for 
local evaluation of programs varying widely in type, size, and purpose. 
The sole restriction on the type of programs considered appropriate was 
that the program's goals were to emphasize student outcomes. 

The second objective facing the Kit staff— and the thrust of this 
paper— was to identify the means by which the Kit would guide principals 
and school personnel through the evaluation processes. As Hull, Kester, 
and Martin (1973) have recently noted, the knowledge base that will assist 
development teams In predicting the response of a target population to a 
particular innovation Is highly limited. At best, development teams can 
rely with a relatively high degree of confidence on a cardinal principle 
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of educational development: Consult with the user. In the case of the 
CSE Formative Evaluation Kit , this meant following the conventional tryout 
and revision cycle with special consideration given to anecdotal data 
gathered through a case-study approach. 

The development strategy 

In order to promote a smooth transition from the development labora- 
tory into the complex dynamics of the school, the designers of the Forma- 
tive Evaluation Kit pilot tested their product by asking principals to 
outline the characteristics they would like to see embodied Into a success- 
fully administered local evaluation product. To help them define these de- 
sired charaterl sties, several cycles of pilot testing were set up. The 
first cycle of pilot testing involved several elementary-school principals 
from neighboring school districts who were given a textbook-style narrative 
about formative evaluation. The narrative was in essence a expanded version 
of the CSE framework for evaluation including suggestions for carrying out 
specific formative evaluation procedures. As anticipated, the panel of 
participating principals was pleased to recognize the germ of a product that 
had the potential of helping them meet state- and district-mandated evalua- 
tion requirements. Conversely, though still foreseen, they were highly 

i 

critical of the "research orientation" of the narrative-style prototype 
which they had reviewed. Their comments served to develop a user-oriented 
and more prescriptive set of materials. This revised version of the 
Formative Evaluation Kit was introduced on a case-study basis into several 
schools for actual use — "a consultant in a book" — for conducting a 



formative evaluation. During this phase of pilot testing, CSE staff members 
were concerned with two dimensions of product efficacy: (1) Did the revised 
product help principals monitor and improve a program while it was in pro- 
gress? and (2) In what way could the procedures advocated in the Kit more 
closely match the requirements of both principals and their instructional 
personnel? Based on how the product performed according to these two dimen- 
sions, the Kit underwent, once again, major revisions in format, tone, and 
direction. There was little resemblance between this revised version of 
the Kit and its earlier version, the textbook-style narrative. CSE staff 
members were keenly aware of the major changes that the refocusing of "re- 
search-based" evaluation to "user-based" evaluation had imposed on the Kit. 
To ensure, however, that the integrity of the original evaluation framework 
had not been lost in this process of evolution, they initiated another tech- 
nical review of the procedures (and their rationale) as prescribed- in the 
Kit . Finally, incorporating all the revisions based on the aggregated data 
from both users and evaluation specialists, the Kit. was launched into the 
field. Forty schools in twenty-seven states are now participating. Data 
currently being collected from user sites via questionnaires and telephone 
interviews are designed to help the CSE staff better understand whether the 
Kit is new satisfactorily meeting the "Intentions and behaviors" of the prln 
cipals and their school staffs as well as help the designers determine 
whether the KU is adequately guiding the users towards the improvement of 
an ongoing program. 
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Some realities of local school evaluation 

The testing cycles just described pinpointed four practical areas of 
concern reflecting the point of view of the user vis a vis local evalua- 
tion of school programs. These four areas are briefly presented below. 

1 . Time 

The reviewer and pilot principals served to underscore a well-known 
notion: Elementary -school principals normally are busy people. 
Principals, as a rule, do not have the time, much less the inclina- 
tion, to become evaluation specialists; in our case, they made it 
abundantly clear that they did not wish to rummage through the typi- 
cally theoretical narratives of many R and D products. At the same, 
time, they wished (or needed) to utilize the expertise and resources 
of their school's own instructional staffs to minimize the burden on 
each principal's time. The optimum situation for conducting an evalu- 
ation evolved from these concerns appeared to be the one In which the 
principal would be able to delegate or share discrete "do-able" tasks 
to staff members. In short, the principals did not want another text- 
book, but rather felt the need for a "cookbook" approach to meeting 
evaluation demands. 

2. Decision orientation 

"Had we but world enough and time..." was the sigh echoed by the re- 
viewer and user-principals Indicating that they. would enjoy specula- 
ting on Che latest evaluation concepts and their applicability to the 
school setting. However, to the Kit staff, it soon became evident 
that principals, in spite of their wistful appreciation of the poet's 
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longings, had neither the 'World" nor the "time" to engage In such theoreti- 
cal speculations. Instead, the principals opted for a brief, attractive pro- 
duct that was highly readable and that could lead easily to specific, measure- 
able tasks, a decision more in keeping with the realities of the local school 
operational cycle. In the local school, the principals were typically re- 
garded as the administrative decision makers; they in turn openly acknowledged 
the teachers' responsibility for much of the inetvuotional decision making; 
the teachers, for their part, would rather leave most of the management-type 
decisions to the appropriate authority, that is, the principal. In matters 
affecting the conditions in which students' learning Is to occur, the instruc- 
tional staff, however, did expect to be consulted. 

3. Technical ease 

» 

In their capacity as administrators, principals generally are neither sta- 
tisticians, nor computer experts, research methodologists, nor educational 
test designers. The reviewer and user-principals emphatically stated that 
they were busy administrators deal ing primarily with matters of school poli- 
cy and operation. Other than for professional growth, the principals clearly 
indicated that under ordinary circumstances, they were not interested in 
using technical ski i is (e.g., statistics, computer programming, or measure- 
ment) to augment their position as decision makers. To the CSE staff, there- 
fore, it became evident that for practical success and feasabiUty, the techni- 
cal characteristics of any product used to conduct an evaluation should be 
kept at a minimum. 
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k. Socio-political concerns 

Principals, as suggested by those involved In the testing of the Kit , 
must answer to many separate, unique (and occasionally, mutually ex- 
clusive) audiences, including teachers, parents and community members, 
and funding agencies. Consequently, any product intended to assist 
with an evaluation of a local school program must be able to balance 
the information needs of these diverse audiences. Most importantly, 
principals must be able to allay any fears on the pari of the school 
staff that formative evaluation is a "judgment" upon individual teach- 
ing skills. Equally important, principals must be able to present to 
parents and the community the rationale supporting a particular educa- 
tional program and explain how a formative evaluation might enable the 
school staff to upgrade that program while it is in operation. And, 
finally, the principal must be able to Justify to funding agencies 
the allocation of resources and provide an explicit record of what has 
transpired during the instructional program. 

From construct to construction 

As described earlier in this paper, the process by which ;he Formative 
Evaluation Kit evolved from a "research-based" output to one that prescribes 
the specific means by which users can conduct an evaluation consistent with 
the CSE framework, did not occur overnight. New features of the Kit were in- 
crementally added and tested as anecdotal and performance data were accumu- 
lated and interpreted. Today, the Kit has reached a relatively steady state 
now that it is in the midst of a national field test. With the wisdom granted 
by hindsight, it is possible systematically to connect current features of the 
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Kit to the realities of local school evaluation as they were uncovered by CSE 
staff members during earlier case studies.. Some of the major revisions made 
in the Kit as a result of these case-study data are summarized in Table 1. 



Table 1 



Summary of Current Features 
in the CSE Elementary School Formative Evaluation Kit 
and Their Relation to Case-Study Data 



NEEDS EXPRESSED BY USER/ 
REVIEWER PRINCIPALS 

1. Time 

'Limited time for conduct 
evaluation 



'Desire to share some eval 
uation tasks with other 
staff members 



2. D ec 1 s j on-0 r ientat j on 

'Need to maintain an adminis- 
trative position that calls 
for action, not deliberation 

'Immediate responsibility for 
administrative decisions such 
as allocation of time and money 



'Ultimate responsibility for 
ensuring adequate conditions 
for instructional program- 
ming 



APPROACH TAKEN IN FCRI1ATIVE 
EVALUATION KIT 

1 . Time 

* Step-by-step format: Division of 
Evaluation into "do-able" tasks allows 
user to proceed one step at-a-time 
without necessity of first reading 
entire Kit 

'Modular formatting: Separate steps 
permit user to accomplish a defined 
task within a short period of time 
or to delegate discrete evaluation 
activities to other staff members if 
appropriate 

2. Dec j s j on-0 r j en ta 1 1 on 

'Measurable steps: Each evaluation 
step (through use of Worksheets) re- 
sults In a specific decision and sug- 
gests ways to implement decisions 

'Resource al location: Several steps 
are devoted to the fair distribution 
and planning of evaluation tasks and 
to keeping financial accounts of funds 
spent on the evaluation 

'Frequent consultation: Steps that im- 
pact upon student learning involve co- 
operative decision making with instruc- 
tional staff 



continued 
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Table 1 (continued) 



USLR NEEDS CSE APPROACH 



'Minimal interest in evalua- 
tion theory 



3. Technical Ease 

'Need for evaluative data 
with high probability of 
being used 



'Extremely limited local 
resources for statistical 
computation and analysis 



'Little or no resources for 
developing measurement in- 
struments 



'Prescriptive tone: Each step focuses 
on ways to apply evaluative princi- 
ples; discussion of underlying ration* 
ale is brief 

3. Technical Ease 

'Evaluation questions: The paradigm 
for generating aseful evaluation 
questions is accompanied by specific 
examples encompass-! ng Implementation 
and progress concerns Emphasis is 
on data* that realist i early used, im- 
prove the program 

'Selection of a suitable measure: An 
INSTRUMENT SELECTOR allows users to 
match the It " eva luat Ion ques t ions to 
8 types of measures. A separate step 
helps the user decide whether to buy 
or to build an instrument designated 
as appropriate by the Selector. 

'Construction of measures: If users 
decide to develop their* own measures, 
the Kit provides an Instrument Con- 
struction Handbook* for the following 
eight types of Instruments: 

CHECKLIST 

SELECTED RESPONSE TEST 

QUESTIONNAIRE 

INTERVIEW 

RECORDS 

PERFORMANCE TEST 
OBSERVATION 

CONSTRUCTED RESPONSE TEST 



cont I nued 
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Table 1 (continued) 



BEST COPY AVAILABLE 



USER NEEDS 



CSE APPROACH 



'Need for more systematic 
methods of analyzing col- 
lected data 



4. Socio- Pol 1 tlcal Concerns 

'Need to inform teachers of 
purpose of evaluation; de- 
sire to involve teachers 
in identification and col- 
lection of data as well as 
in making key decisions 



'Desire to communicate with 
parents and community mem- 
bers about program goals 
and evaluation 



'Need to report to funding 
agencies 



'Review of data: Kit gives instruc- 
tions for summarizing collected data 
according to the given Evaluation ques- 
tions. Emphasis is on use of simple 
tables and figures. Modified Delphi 
techniques are incorporated into a step 
in which the principal and staff make 
recommendations for program modifica- 
tion according to the data 

k, Soc to* Pot j t tea 1 Concerns 

'Staff involvement: Kit suggests that 
the instructional staff participate in 
major decisions regarding the effective- 
ness of the program; sample meeting 
agendas and memoranda are provided. 
Users are encouraged to explore the in- 
formal sources of information about the 
program through discussion with staff 
and observation of side-effects 

'Overhead transparencies: A visual pre- 
sentation that can be adapted by the 
principal to the specific program. The 
presentation encompasses both the in- 
structional components of the new pro- 
gram and the formative evaluation 

'Report formats: One step of the Kit 
Hives alternate report formats and 
suggests information that could be in- 
corporated into a report to a funding 
agency 



Highlights and Summary 

Most decisions regarding format, tone, and level of specificity of the 
Kit were influenced by the principals ' expressed need for an efficient, 
action-oriented package. Due to their busy schedules, they preferred con- 
cise, attractive, prescriptive materials that led directly to specific 
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accomplishments. Thus, In the Kit each discrete aspect of the evaluation 
consists of a short DISCUSSION, followed by Instructions for APPLICATION 
to the local program. In addition, the design of the Kit allows the user 
to proceed one step at a time, with each step resulting In a decision (e.g., 
the Identification of Evaluation questions) or In an action (e.g., the col- 
lection of data). The Kit suggests that one person in the school assume 
the role of Evaluator. While the Evaluator is to coordinate use of the 
K]£ (and thereby manage the evaluation), some steps can be carried out by 
other staff members or at regular faculty meetings. In this way the princi- 
pal, who would most likely be the Evaluator, has the option of defining his 
role as leader of an evaluation team. Provision is also made for keeping 
track of the money spent on evaluation activities and for effectively 
allocating tasks, conscious of the inexorable operation of Murphy's Law, 
"If anything can go wrong, it Willi" The modular format and prescriptive 
approach of the JOt represents the JUt staff's attempt to ease, not In- 
crease, the administrative problems of the principal. 

It should be noted here that the most difficult aspect of the develop- 
ment process was to operational ize the theoretical framework on which the 
Kit, was based. The pilot-test data readily showed that elementary schools 
typically have limited technical resources for statistical computation and 
analyses, for Instrument development, or for computer programming. Further, 
the collected data suggested that the evaluation design itself must remain 
simple for two reasons: (1) because the Kit, is most likely to be used to 
evaluate a small program (one or two classrooms), (2) because most schools 
are unable to implement random assignment of students to treatment or control 
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groups of any type. This situation is not as bleak as it sounds. Because 
they can be characterized as in-house.Evaluators, principals who do use 
the KH possess certain types of knowledge and authority which an outside 
professional Evaluator, however competent, cannot possess. For example, 
a principal is not only familiar with the stated goals of a program, he 
also is aware about unstated expectations of teachers and parents. This 
special knowledge puts him Into an excellent position for formulating eval- 
uation questions that take into account some of the more subtle aspects of 
program operation, such as parent and teacher satisfaction with program 
processes and questions relating to student achievement. Also, the prin- 
cipal is In a position^ receive constant feedback from his staff, for- 
mally and informally, about successes and failures of the program. Finally, 
the principal is in a perfect situation to oversee the implementation of 
recommended changes. 

Taking ail these facts Into account, Klt^ staff members realized that 
a local elementary school has available a wealth of resources that can be 
channeled towards the conduct of an effective evaluation. Thus, rather 
than offering a highly theoretical — and technical — view of evaluation, 
the Kit presents formative evaluation as a process of asking pertinent 
questions about program operation (including questions about proper instal- 
lation of program components, about progress towards the stated goals, and 
about unanticipated outcomes or (side-effects'*); gathering and interpreting 
data to answer the Evaluation questions; and making changes in the program 
where appropriate. To carry out these procedures, the KTt provides practi- 
cal guidelines to help principals and their staffs apply their working 



knowledge of the program In a systematic manner. Most steps of vhe Kit 
Include WORKSHEETS for record tng dec ts tons that have been or need to be 
made; all steps contain an APPLICATION Section which gives specific ex- 
amples and instructions. The Appendix to this paper presents the objec- 
tive for each of the 15 Evaluation Steps which together, comprise the Kit. 

There are various additional inclusions in the Kit , such as numerous 
technical aids to assist the principal when it ts found that adequate 
local resources are lacking; an INSTRUMENT SELECTOR directs him to appro- 
priate measurement devices for gathertng data whtch will enable him to 
answer the Evaluation questions generated In an earlier step. In addi- 
tion, should the user find available published instruments inappropriate, 
the Kit, includes a separate INSTRUMENT CONSTRUCTION HANDBOOK which pro- 
vides instructions, geared to the layman, for developing each of the 8 
types of instruments found on the Selector. Other steps of the Kit offer 
concise instructions for collecting data and for displaying findings with 
simple Tables and Figures. 

Kit staff members also discovered that although evaluation has be- 
come a fifty-cent word among educational policy makers, to numerous teach- 
ers the term hints at reduced classroom autonomy; and to parents, the word 
may well connote the suspect "educational experimentation." To allay these 
fears on the part of the teachers, at major decision points during the for- 
mative evaluation (such as the generation of pertinent evaluation questions 
or the recommendation of program changes) the Kit suggests that the Evalua- 
tes Involve staff members in cooperative decision making. The setting for 
such decisions can be a regular faculty meeting or Informal discussion with 
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individual staff members. For the parents, the Kit includes short visual 
presentation with overhead transparencies as a vehicle to facilitate com- 
munication with parents and community members about program goals and the 
evaluation. As for the funding agencies, the final step cf the Kit presents 
a format for setting up reports to funding agencies, an audience of undoubted 
importance to many principals. 

To end on a philosophical note: it has been said that the world is 
full of many problems — and an even greater number of solutions. For 
educational developers who, by the nature of their profession, tend to be 
pragmatic, the "best 11 solution to any problem is the one that is Implemented . 
In this spirit, the staff of the Formative Evaluation Kit sought the means 
to help elementary- school principals and their staffs monitor and evaluate 
the progress of a local school program. Perhaps, it is by matching means 

i 

to ends that solutions to some educational problems may, in fact, be imple- 
mented. 
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Section I: Ascertaining Decision Areas of Concern 



Step One: 
Step Two: 
Step Three; 



Reviewing the Program Plan 
Constructing a Master Plan 
Developing Evaluation Questions 



Step Four: 
Step Five: 
Step Six: 
Step Seven: 



Section II: Selecting Appropriate Information 

Selecting a Measurement Strategy 
Estimating Evaluation Resources 
Determining Appropriate Measures 
Setting Evaluation Deadlines 



Section Ml: Collecting and Analyzing Information 



Step Eight: 
Step Nine: 
Step Ten: 
Step Eleven: 
Step Twelve: 



Establishing Communication 
Collecting Data 

Discovering Unanticipated Outcomes 

Scoring Measures 

Preparing Data for Presentation 



Step Thirteen: 
Step Fourteen: 
Step Fifteen: 
Step Sixteen: 



Section IV: Reporting Data 

Making Recommendations 
Implementing Recommended Changes 
Preparing Informal Reports 
Preparing Formal Reports 
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